There is no root page on the site since all four domains are mapped to a Group page. As for selecting the starting node, we had issues with this also. Like I mentioned before, in order to access the specific group page in the dialog, for selecting the starting node, we had to access the site via localhost or some other non-mapped domain. Otherwise, the page listing in the dialog is built off of the current domains mapped page, which in selecting "All Pages", returns an empty string. This empty string causes the site to index all pages.
Here is an example of what is setup and then a test I ran that simulates the problem.
web.config =>
<urlMappings>
<add key="domain1.com"
value="domain1"
shared="false" />
<add key="domain2.com"
value="domain2"
shared="false" />
<add key="domain3.com"
value="domain3"
shared="false" />
</urlMappings>
Sitefinity Admin => Pages
All Pages -> //root of the site
domain1 (domain1.aspx) //Setup as a group page
-> sub page
domain2 (domain2.aspx) //Setup as a group page
-> sub page
domain3 (domain3.aspx) //Setup as a group page
-> sub page
Sitefinity admin => Administration/Services/Search
1. Create an index and named it "Test"
2. Add a PageIndex -> Set auto index to true, Index Non-public pages to false, starting node url to ~/domain1.aspx
4. Save and run "Start Indexing now". Results show as index x number of words and 3 pages
This is a mock setup of what we had when I submitted this post. After your reply, I looked through the code to try and understand the domain mapping a bit and came to the conclusion that if the starting node url is the same as a mapped page, then the value returned to the indexing provider is the root or the "All Pages" for the whole Sitefinity site. So I thought of a way to test this. I changed the value of the mapping entry for domain1.com from domain1 to domain11. I then went back to the index "Test" and ran the "Start Indexing Now". This time the results came back with indexed x words and 1 page.
Since the site is setup with just the three pages, which are all page groups, in the root of the site and these pages are mapped to the domain mapping in the web.config, we are getting indexes with all pages. If the starting node url for the index is not mapped via domain mapping, then the index will function in the manner you described. It is only when the page is mapped via domain mapping that the index is returning all pages in the site.
Richard