Walking from the Botanic gardens, I took a shortcut path to reach Circular Quay wharf across the "Memory is Creation Without End" relics.

Start Point: Circular Quay Wharf.

One can also take a train to the Circular Quay station to reach the Circular Quay wharfs.

https://transportnsw.info/routes/ferry will provide information on the various ferry services available and the timings of the ferry departures. The digital sign-boards at near the Wharfs also have information on which number wharf serves which location and the departure timings.

The Manly ferry departs from Wharf No. 3
Customs House at Circular Quay
You get a wonderful view of the Opera House as the ferry departs from Circular Quay. 
Sydney CBD Skyline from the ferry

The best views of Sydney harbour bridge and the Opera house are from the public ferries.

Sydney Harbour Bridge
Manly Fast Ferry departing from Manly Wharf

Once you get down from the ferry at Manly, head towards Stand D across the road for bus No. 135. For schedule of the bus, it's best to check Google maps which is pretty accurate in showing the next bus timings.

The bus route for 135 - https://transportnsw.info/routes/details/sydney-buses-network/135/28135

Even though the route map shows that the bus goes all the way to Fairfax walk, it does not and the last stop is the Q Station stop. The rest of the distance, I just walked on the road instead of the walking path.

Note: There is no footpath from the last bus stop to Fairfax lookout.

North Fort:

Strategically placed at the northern entrance to Sydney Harbour, North Fort was part of a defence system that spanned 300km of coastline during World War II.

The once active army base previously featured 9.2 inch calibre guns, capable of firing a distance of 26km. Each heavy weapon was operated by nine soldiers with support from personnel running the underground shell and engine rooms.

North Fort was built to protect Sydney from air and sea attack, completed around 1938, playing a pivotal role as an enemy deterrent during World War II. North Fort is now the Royal Australian Artillery National Museum enabling visitors to enjoy the tunnels, guns, scenery and BBQs, or perhaps relax in the café with spectacular views of Sydney Harbour. North Fort gates open from 6am – 8:30pm (October – April) and 6:30am – 6:30pm ( May – September).

With its large network of gun emplacements and underground tunnels, North Fort was a crucial part of Sydney’s coastal defence system throughout World War II.

North Head:

The heath-covered clifftops at North Head reveal unending ocean views and a spectacular panorama of the harbour and Sydney skyline. Situated on the northern-most edge of the harbour within sight of Hornby Lighthouse on South Head, it’s a relaxing ferry ride from Circular Quay. One of those things to do when visiting Sydney, it’s near to Manly, and a great day trip for locals and tourists alike.

North Head is home to an endangered population of Long-nosed Bandicoots, once common throughout Sydney. It is also a refuge for echidnas, reptiles, frogs and more than 100 species of bird, including the New Holland Honeyeater, the Rainbow Lorikeets and the Little Wattlebird. Recently, locally extinct species such as the Eastern Pygmy Possum, Brown Antechinus, and Bush Rat were re-introduced to the area.
North Head Sanctuary was the backdrop for some of the earliest interactions between Aboriginal People and Europeans, and the area was also used to quarantine Australia from deadly epidemic diseases. During the Second World War, North Head was one of the most heavily fortified sites in Australian history. Remnants of those fortifications can be explored today.
Wildlife! 
Beautiful view from the lookout point!

Walking further ahead will take you to the Fairfax walk/trail.

No footpath! Be careful of cars while walking!
The Fairfax lookout trail
View of Sydney CBD Skyline from Fairfax lookout!
Dangerously swirling water at the bottom of the cliffs!
Dangerously swirling water at the bottom of the cliffs!
Beautiful sunset! 
The Manly public ferries can be seen in the picture!
Amazing view of the North Head cliffs!
Fairfax Walking Track
Fairfax walk
North Head Artillery Lookout
View from inside!
The beautiful path overlooking the sea!
North Head Artillery Lookout

Third Quarantine Cemetery:

Established in 1881 for victims of a smallpox epidemic, the Third Quarantine Cemetery at North Head sanctuary, Manly, is included on Australia’s National Heritage List as a significant example of the nation’s evolving quarantine practices.

By the time of the cemetery’s closure in 1925, more than 240 people had been interred there, having succumbed to ravages including influenza, the bubonic plague, smallpox and scarlet fever.

Third Quarantine Cemetery

Q Station:

The Quarantine Station was established primarily to regulate the risk of disease importation through the migration of free and convict Europeans, and the arrival of merchant shipping. There was always a close link between the requirement for quarantine and the ebb and flow of sea-borne immigration; and the growth of the Quarantine Station from the 1830s parallels the changes in immigration policy and practice. The other major influence was the imperative to limit disruption to the increasingly commercially-sensitive shipping industry.

The North Head Quarantine Station is a heritage-listed former quarantine station and associated buildings that is now a tourist attraction at North Head Scenic Drive, on the north side of Sydney Harbour at North Head, near Manly, in New South Wales, Australia.

I took the bus 135 back to Manly wharf and took the ferry back to Circular Quay.

View of the Sydney Opera house and CBD from the ferry
Sydney Opera House
Sydney Harbour Bridge
View of the harbour bridge from the Circular Quay wharf. 
Pitt Street Mall
Sydney Town Hall.