In 1932, under the reign of Emperor Bao Dai, the house of the Royal Guard Office was constructed within the Ngu Vien Garden. Ngu Vien Garden was at the Northeast of Hue forbidden Purple City, constructed in the early Emperor Minh Mang’s reign (1820). The architecturalconstructions in this garden were done sophisticatedly. It was considered a royal flower garden, where emperors often arrived at for entertainment and making poems. Ngu Vien Garden was classified as the fifth beautiful landscapes of Hue Ancient Capital City in Emperor Thieu Tri poem anthology. However, from the reign of Emperor Thanh Thai (1889 - 1907), the architecture of Ngu Vien Garden was changed a lot; some constructions were removed; and the landscape was abandoned.
Under the reign of Emperor Bao Dai, the Royal Guard Office played the role of Cabinet appeared from the reign of Emperor Minh Mang.