Many employers have probably had similar thoughts from time to time, but few of them would have been as accurate about their domestic staff's superior skills. What Pickering did next was highly unorthodox: he got rid of his team of male staff and employed his "Scotch maid", a single mother from Dundee named Williamina Fleming, instead. This somewhat irregular method of hiring staff, to analyse photographs he was taking of the night sky, proved to be particularly successful.
From this chance beginning, Fleming, who was known as Mina, would go on to become one of the most accomplished astronomers of all time. She became noted for discovering exploding stars, identifying ten of the 24 novas that were known to science by the time of her death, and achieved considerable recognition in the field.
She led a team of women - known in the male-dominated world of the late 19th century as "Pickering's Harem" - that made a number of significant breakthroughs, chief among them a way, for the first time, of measuring the distances of stars.
In 1907 Fleming published a list of 222 variable stars she had discovered, prompting one British astronomer to comment: "Many astronomers are deservedly proud to have discovered one... the discovery of 222 is an achievement bordering on the marvellous."
By 1911, the year she died of pneumonia at the age of 54, she was a member of the Royal Astronomical Society as well the astronomical societies of America, Mexico and France. A crater on the moon was even named in her honour.
International best-selling science writer Dr Simon Singh, who will speak about the universe and Fleming's contribution to our knowledge of it in Dundee tomorrow night, has looked into her remarkable life story and believes this "Scotch maid" is one of both Scotland's and astronomy's unsung heroes: "At a time when women were not supposed to be scientists, she made remarkable breakthroughs.
"In science, data sets are invaluable. They allowed you develop ideas. In this case the data sets meant photographs. The women were only supposed to perform dull number-crunching, but they were very bright so they drew their own conclusions and made several important discoveries."
Dundee-born Stevens had left school at 14 to become a student teacher, then emigrated to America at the age of 21 with her new husband, James Fleming.
But only a year later in, 1879, her life was plunged into crisis when she was abandoned by her husband while pregnant with their child.
This abrupt change in fortunes forced her to take a job as a maid in Pickering's house. The future may have seemed bleak, but it was about to change dramatically for the better.
It is clear that the astronomer made a considerable impression on the pregnant Fleming, as when her son was born - in autumn 1879 on a brief trip to Scotland - she named him Edward Pickering Fleming.
The records show that two years later she was hired to work at Harvard Observatory as a permanent employee, doing mathematical calculation and clerical work, shortly after Pickering's famous tirade against his male staff.
However, despite Pickering's decision to employ women as "compuiters" - a term originally used to describe people who manipulated data and performed calculations - there were limits to this early version of male feminism.
The women were not actually allowed to stay up at night and make observations. They were instead restricted to analysing the pictures taken by Pickering, who was a pioneer of astronomical photography.
And, at the turn of the century, when Fleming had established her reputation, she was working 60-hour weeks for $1,500 a year, far less than a newly employed male assistant. Fleming, who was paying for her son to study at the Massachusetts Institute of Technology, was by no means happy about the pay gap, and expressed her annoyance in her diary.
Among her notable achievements - despite having no formal training in astronomy - was a way of classifying stars into 17 different types, which was later named the Pickering-Fleming System. In 1886, she was in charge of a project to classify thousands of stars by the spectrum of light they produced. Four years later, the Draper Catalogue of Stellar Spectra was published, detailing 225,000 stars in nine volumes. Fleming discovered a new and easy way of identifying stars of varying brightness, which proved vital to astronomy, and also selected stars that shone with a constant light to act as comparisons, developing "the first photographic standard for determining the magnitude of star brightness".
She also discovered 94 of the 107 Wolf-Rayet stars known at the time; these are "superluminous stars" in which helium, rather than hydrogen, plays the bigger part. In 1898, in recognition of her considerable work, she was appointed curator of astronomical photographs at Harvard. In 1910 she published a paper detailing her discovery of "white dwarfs", very hot stars that tend to have a bluish-white colour, which are thought to be stars in the final stages of their existence.
Perhaps her biggest role, though, was as a trailblazer for women in astronomy. Fleming had been put in charge of the female computers of Pickering's Harem in the mid-1880s and also become the editor of all the observatory's publications - a role she disliked, because it got in the way of real astronomy.
Then, in 1893, at the Chicago World's Fair, she made a landmark speech about women working in astronomy. It prompted other observatories to open their doors to what had previously been a male preserve.
This may well have made an impression on Henrietta Leavitt, a profoundly deaf woman in her mid-twenties who had graduated from Harvard University the year before and who went on to become the most famous of all her contemporaries.
She volunteered to work at the observatory and, after being formally hired by Fleming, spent years looking at photographic plates for variable stars. She discovered more than 2,400 - about half of all those known at the time - prompting Princeton University professor Charles Young to describe her as "a variable-star fiend".
Leavitt found herself obsessed with stars known as Cepheid variables. After months of working on them, she hit on the astonishing revelation that it was possible to work out the true brightness of these stars by measuring the speed at which they moved from the brightest stage to the dimmest and back again.
This was a staggering development. Prior to this it was impossible to tell whether a prominent star in the sky was actually quite dim and only appeared to be brighter than others because it was nearby or whether it was incredibly bright and far away.
Being able to tell the true brightness of the Cepheids meant that astronomers were able for the first time to say how far away they were.
This groundbreaking discovery enabled astronomers to show that all stars are moving away from each other - evidence that led to the Big Bang theory of the creation of the universe.
Another notable member of the team was Annie Jump Cannon who, on the death of Fleming, succeeded her and later became the first woman to be awarded an honorary doctorate by Oxford University.
She came up with a new way of classifying stars that is still in use today.
Cannon, who was also deaf, divided the stars into seven classes: O, B, A, F, G, K and M. Perhaps continuing an old-fashioned tradition that started because of Cannon's gender, astronomy students today remember this using a mnemonic: "Oh, Be A Fine Girl - Kiss Me!"
The success of these women and others among the computers can perhaps be partly put down to the example set by Fleming and the way she led the team. At Harvard, she was said to have been "remembered by her friends and colleagues as a person with a highly magnetic personality and attractive countenance".
Today she is remembered in the Encyclopaedia Britannica's profiles of the 300 women who changed the world, alongside the likes of scientist Marie Curie, French intellectual Simone de Beauvoir, human rights campaigners such as Emmeline Pankhurst and Rosa Parks, and such great historical figures as Joan of Arc and Cleopatra.
Dr Singh will speak today about the Big Bang and the Universe at St Andrews University at 1pm and at Glasgow Science Centre at 6:30pm. He will speak at Dundee University at 6pm tomorrow.