The phrase has its source in the prejudice that people from the English West Midlands city of Birmingham (Brumagem, or Brummagem) were unsophisticated - a prejudice that persists in the UK today. Other places, notably Liverpool and Ireland, have suffered the same jibe. The 'Brummagem screwdriver' is really just a weak joke at the expense of Brummies, implying that they are such poor and lazy workers that they knock screws in with a hammer. In fact, the Birmingham and Black Country area has long been a source of excellent engineers who staffed the now sadly diminished car industry and other advanced metalworking concerns.
The city of Birmingham, then a small town, is listed in the Domesday book as Bermingeham. Brumagen is one of the versions of the name. There are numerous such variant spellings and pronunciations, for example, Bernynghem, Birmingecham, Bromwicham, Brummindgeham - Chinn and Thorne's Proper Brummie dictionary refer to over 140 such variants.
Whether Brumagem derived as a deliberate mispronunciation of Birmingham or whether both names derived from the original Bermingeham is unclear. In Middle English many places had several names and little attempt was made to determine a 'correct' pronunciation or spelling. Those living in Birmingham and the Black Country, will be used to hearing the name pronounced as 'Birmigum'.
The Bromwicham form persists in the name of the town to the west of Birmingham - West Bromwich.