ANDROID: mm: introduce vma refcounting to protect vma during SPF
Current mechanism to stabilize a vma during speculative page fault
handling makes a copy of the faulting vma under RCU protection. This
makes it hard to protect elements which do not belong to the vma but
are used by the page fault handler like vma->vm_file.
The problems is that a copy of the vma can't be used to safely
protect the file attached to the original vma unless the file is
also released after RCU grace period (which is how SPF was designed
originally but that caused performance regression and had to be
changed).
To avoid these complications, introduce vma refcounting to stabilize
and operate on the original vma during page fault handling. Page
fault handler finds the vma and increases its refcount under RCU
protection, vma is freed after RCU grace period, vma->vm_file is
released only after refcount indicates no users. This mechanism
guarantees that once get_vma returns a vma, both the vma itself and
vma->vm_file are stable.
Additional benefits of this patch are: we don't need to copy the vma
and no additional logic is needed to stabilize vma->vm_file.
Bug: 257443051
Change-Id: I59d373926d687fcbd56847a8c3500c43bf1844c8
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
diff --git a/mm/memory.c b/mm/memory.c
index 17a03bf..6bc34d7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -209,6 +209,35 @@ static void check_sync_rss_stat(struct task_struct *task)
#endif /* SPLIT_RSS_COUNTING */
+#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
+
+struct vm_area_struct *get_vma(struct mm_struct *mm, unsigned long addr)
+{
+ struct vm_area_struct *vma;
+
+ rcu_read_lock();
+ vma = __find_vma(mm, addr);
+ if (vma) {
+ if (vma->vm_start > addr ||
+ !atomic_inc_unless_negative(&vma->file_ref_count))
+ vma = NULL;
+ }
+ rcu_read_unlock();
+
+ return vma;
+}
+
+void put_vma(struct vm_area_struct *vma)
+{
+ int new_ref_count;
+
+ new_ref_count = atomic_dec_return(&vma->file_ref_count);
+ if (new_ref_count < 0)
+ vm_area_free_no_check(vma);
+}
+
+#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */
+
/*
* Note: this doesn't free the actual pages themselves. That
* has been handled earlier when unmapping all the memory regions.