In 1841 the family cannot (yet) be found.

In the 1851 census the family was:

                                Marital
First name(s)  Last name  Role  status  Gender  Age  Birth year  Occupation  Birth place

Richard L  Adams    Head  M  M  49  1802  Perpetual Curate  London, Middlesex
Elizabeth  Adams    Wife  M  F  44  1807  Peers Daughter    Northamptonshire
Henry R    Adams    Son   U  M  17  1834  Scholar           London, Middlesex
Fanny N    Adams    Daur  U  F   4  1847  Infant            Wimbledon, Surrey
Emma       Sparrow  Serv  M  F  27  1824  Nurse             Sussex
Martha     Heffer   Serv  U  F  24  1827  Ladys Maid        Bristol, Gloucestershire
Elizabeth  Tudor    Serv  U  F  45  1806  Cook              Shrewsbury, Shropshire
Ursula     Futcher  Serv  U  F  36  1815  Housemaid         Downton, Wiltshire
William    Brown    Serv  M  M  57  1794  Butler            Birmingham, Warwickshire
John       Tibb     Serv  U  M  14  1837  Page              London, Middlesex
Elizabeth  Barnes   Serv  U  F  16  1835  Kitchenmaid       Wimbledon, Surrey
Richard    Adams    Son   U  M  13  1838  Scholar           Bath, Somerset

Ref:
Archive reference  HO107
Piece number       1603
Folio              42
Page               1
Combines:          1851 ref HO107, piece 1603, folio 42, p. 1

It is odd that her second son is listed last, among the servants.  Perhaps he had been at a boarding school and reappeared at the last moment?
_________________

In 1861 Elizabeth was not to be found but the household was:

                                Marital
First name(s)  Last name  Role  status  Gender  Age  Birth year  Occupation  Birth place

Richard L  Adams    Head        M  M  59  1802  Rector Of Shere  St Geo Han Sq, London
Charles V  Adams    Son         U  M  25  1836  Curate Of Shire  St Geo Han Sq, London
Louisa     Ross     Adopt Daur  U  F  14  1847  Scholar          At Sea
Mary C     Bennett  Governess   U  F  23  1838  Governess        Burringham, Warwickshire
Jane       Sellars  Serv        U  F  26  1835  Parlourmaid      Nottinghamshire
Frances    Sheare   Serv        U  F  26  1835  Cook             Arden, Sussex
Elizabeth  Ranger   Serv        U  F  36  1825  Housemaid        Albury, Surrey
Julia      Crawley  Serv        U  F  15  1846  Housemaid        Huntingdonshire
Maria      Grant    Serv        U  F  12  1849  Kitchenmaid      Shere, Surrey
William    Falkner  Serv        U  M  14  1847  Page Boy         Chichester, Surrey

Ref:
Archive reference  RG09
Piece number       426
Folio number       42
Page number        16
Combined:          1861 ref RG09, piece 426, folio 42, p. 16

Elizabeth is not to be found in the 1861 census.  There is only one Elizabeth Adams shown who was born in Northants in 1807 plus or minus 2 years on FindMyPast and this was not her.
__________________

In 1871 they were on their own in Shere, Surrey:

Name                Role  Sex   Age  Birth  Year  Occupation  Where Born

ADAMS, Richard L    Head  M  69  1802  Middlesex
ADAMS, Elizabeth A  Wife  F  64  1807  Northamptonshire, Lilford
SPARROW, Emma       Serv  F  45  1826  Sussex
MONK, Mary          Serv  F  56  1815  Middlesex
HALLIDAY, Mary A    Serv  F  23  1848  Berkshire
SPARKES, Amy        Serv  F  35  1836  Berkshire
WESTON, Elizabeth   Serv  F  36  1835  Hampshire
EDSER, Walter Serv  M  19  1852  Surrey

1871 ref: RG10, Piece 810, Folio: 36, p. 3    
  
_______________

In 1881 she and Emma Sparrow (see above in 1851 and 1871) were in a dwelling, possibly within a boarding house, in Tormoham with Torquay, Devon; she called herself a Peers daughter and was down as born in Lilford, Northamptonshire in 1807.
_______________

Her children from the above censuses (first three from 1851, fourth from 1861):
5First                     Marital         Birth
name(s)  Last name  Role  status  Gender  year  Occupation        Birth place
Henry R    Adams    Son   U       M       1834  Scholar           London, Middlesex
Charles V  Adams    Son   U       M       1836  Curate Of Shire   St Geo Han Sq, London
Richard    Adams    Son   U       M       1838  Scholar           Bath, Somerset
Fanny N    Adams    Daur  U       F       1847  Infant            Wimbledon, Surrey

Additionally in In 1881 there was an Ernest Powys Adams at Marlborough school, aged 15 and born at North Curry, Somerset.  Might this be a son of Henry R born in 1834 or Charles V born in 1835?
________________

While I have found Charles Vernon Adams on several records, I cannot find either Henry R or Fanny N or Richard on any identifiable records.
________________

I had their marriage down as 15 Feb 1828 but this is unbelievable.

Yet here is the notice in the Stamford mercury of Fri 22nd of Feb 1828, on p. 4:

“MARRIED:  The Rev L Adams, of Grosvenor Place to the Hon Eliza Atherton Powys, fourth daughter of the late Lord Lilford.”